The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology.
In general, infertility is defined as not being able to get pregnant (conceive) after one year (or longer) of unprotected sex. In vitro fertilization (IVF) is a medical treatment option for a significant population of couples experiencing infertility. Infertility can arise from a combination of disorders including male factor causes and female causes with tubal blockage, decreased egg number, decreased egg quality, ovulatory disorders, endometriosis, pelvic adhesions, and unexplained causes. The most aggressive form of treatment of infertility is called assisted reproductive technology (ART) which specifically means technology in which ova (i.e., egg cells) are extracted from the woman's ovaries, fertilized outside of the body, and the resultant embryo is transferred back into the uterus of the patient. The goal of ART is to identify the best embryo or embryos and return it to the patient's womb. A fundamental part of success in ART is creating the best quality eggs possible for an individual patient. The quality and maturity of the eggs is directly predictive for the likelihood the eggs fertilize to become embryos and for the quality of the embryos. The quality of the resultant embryo is predictive for the implantation rate which is defined as (the pregnancy per one embryo transfer) and the overall pregnancy rate (if a multiple embryo transfer is performed). The thickness and developmental pattern for the endometrium (the inner lining of the uterus and location for embryo implantation) is also predictive of implantation and pregnancy rates.
Ovarian follicles contain oocytes which are surrounded by granulosa cells. There are four different types of follicles at distinct stages of development: primordial, primary, secondary and tertiary (or antral). The number of primordial follicles, which is the true ovarian reserve, is determined in the fetus and declines throughout a woman's life. Primordial follicles consist of a dormant single layer of granulosa cells surrounding an oocyte. They are quiescent, but initiate growth depending on a sensitive balance between the factors that promote proliferation and apoptosis (i.e., cell death). When changing to primary follicles, the granulosa cells start to duplicate and become cuboidal. A glycoprotein polymer capsule, the zona pellucida, forms around the oocyte, separating it from the granulosa cells. When becoming secondary follicles, stroma-like theca-cells that surround the outer layer of the follicle undergo cytodifferentiation to become theca externa and theca interna, which are separated by a network of capillary vessels. The formation of a fluid-filled cavity adjacent to the oocyte, the antrum, defines the tertiary, or antral, follicle. Since there is no test available to evaluate the true ovarian reserve, ovarian antral follicle count (AFC) is accepted as a good surrogate marker. Ovarian antral follicles can be identified and counted using transvaginal ultrasound (US). AFC is frequently assessed in women of reproductive age, for various reasons including predicting the risk of menopause, suspicion of ovulatory dysfunction secondary to hyperandrogenism anovulation, and work-ups for infertility and assisted reproduction techniques.
Ultrasound (US) imaging has become an indispensable tool in the assessment and management of infertility for women undergoing ART. Decreased ovarian reserve and ovarian dysfunction are a primary cause for infertility, and the ovary is the most frequently ultrasound-scanned organ in an infertile woman. The first step in an infertility evaluation is the determination of ovarian status, ovarian reserve and subsequent follicle monitoring. Ovarian antral follicles can be identified and manually counted using transvaginal US. The antral follicles become more easily identifiable by US when they reach 2 millimeter (mm) in diameter, coinciding with the attainment of increased sensitivity to follicle-stimulating hormone (FSH). Antral follicles measuring between 2 and 10 mm are “recruitable,” while antral follicles greater than 10 mm are usually referred to as “dominant” follicles. Infertility can also be associated with the growth of a dominant follicle beyond a preovulatory diameter and subsequent formation of a large anovulatory follicle cyst. The ovary is imaged for its morphology (e.g., normal, polycystic, or multicystic), for its abnormalities (e.g., cysta, dermoids, endometriomas, tumors, etc.), for its follicular growth in ovulation monitoring, and for evidence of ovulation and corpus luteum formation and function. Ovulation scans enable the physician to determine accurately the number of recruitable eggs, each individual follicle's egg maturity, and the appropriate timing of ovulation. In general, during infertility treatment, frequent two-dimensional (2D) US scans are done to visualize the growing follicles, and measurements are made of all follicles in the ovary (customarily 15 to 20 follicles) to determine the average follicular size of each follicle. This is performed 4 to 6 times during the 10 days while a patient is on medications (e.g., gonadotropin therapy). The typical time required to perform the ultrasound is approximately 10 to 15 minutes per patient plus additional time to enter the data into an electronic medical record (EMR) system (approximately 5 minutes) or electronic health record system (HER). Ovaries are classified into three types based on the number and size of the follicles. A cystic ovary is one containing one to two follicles measuring greater than 28 mm in diameter. A polycystic ovary is one containing twelve or more follicles measuring less than 10 mm. An ovary containing one to ten antral follicles measuring 2-10 mm and one or more antral follicles measuring 10-28 mm size, the “dominant” follicles, F is considered a normal ovary.
Current 2D US measurements of the follicles are made under the assumption that they are round, but frequently the follicles are irregularly shaped, making the measurements inaccurate. There is also significant human variability in measuring millimeter dimension objects by US, further complicating the accuracy of using this modality for follicle monitoring. It is also difficult to identify all of the follicles in the ovary using 2D US, leading to frequently missed measurements. The last complexity, but not the least, is the inter-observer follicle size measurement variabilities of ultrasonographers which requires further scrutiny by physicians during review. With the advent of three-dimensional (3D) ultrasound, resolution has steadily improved along with data connectivity. 3D ultrasound measurements of the ovary are performed by simply placing the probe in the vagina, directing it to the ovary, and pushing a button. 3D-US imaging has the advantage of a shorter examination time, as it enables storage of acquired data for offline analysis, and better inter-observer reliability. However, new features such as automated volume calculation (SonoAVC; GE Medical Systems) technique can incorrectly identify adjacent follicles and extraovarian tissue as being only one follicle. Despite improvements, there is no consensus on the best US technique with which to perform follicle counting. All semi-automated methods currently available have pros and cons and are affected by the operator's preference and skill, which are prone to inaccuracies and variability.
From a digital data processing systems perspective, the follicles are the regions of interest (ROIs) in an ovarian ultrasound image and can be detected using image processing techniques. The basic image processing steps, namely, pre-processing, segmentation, feature extraction and classification, can be applied to this complex task of accurate follicle recognition. However, imaging modalities that form images with coherent energy, such as US, suffer from speckle noise, which can impair the performance for automated operations such as computer aided diagnostics (CAD), a system that can, for example, differentiate benign and malignant lesion tissues for cancer diagnosis. CAD, in the context of ART, is desirable to address the tedious and time-consuming nature of manual follicle segmentation, sizing, counting, and ovarian classification, where accuracy requires operator skills and medical expertise. In the image classification process, a task is to specify the presence or absence of an object; the task of counting the objects also requires reasoning to ascertain the number of instances of an object present in a scene.
Speckle (acoustic interference) refers to the inherent granular appearance within tissues that results from interactions of the acoustic beam with small-scale interfaces that are about the size of a wavelength or smaller. These non-specular reflectors scatter the beam in all directions. Scatterings from these individual small interfaces combine through an interference pattern to form the visualized granular appearance. Speckle appears as noise within the tissue, degrading spatial and contrast resolution but also giving tissues their characteristic image texture. The speckle characteristics are dependent on the properties of the imaging system (e.g., ultrasound frequency, beam shape) and the tissue's properties (e.g., scattering object size distribution, acoustic impedance differences). Speckle is a form of locally correlated multiplicative noise, which may severely impair the performance of automatic operations like classification and segmentation, aimed at extracting valuable information for the end user. A number of approaches have been proposed to suppress speckle while preserving relevant image features. Most of these approaches rely on detailed classical statistical models of signal and speckle, either in the original or in a transform domain. The need exists for alternative methods to improve US resolution for improving AFC accuracy and CAD for ART.
The emerging field of machine learning (ML), especially deep learning, has made a significant impact on medical imaging modalities. Deep learning (DL) is a new form of ML that has dramatically improved the performance of machine learning tasks. DL uses artificial neural networks (ANNs), which consist of multiple layers of interconnected linear or non-linear mathematical transformations that are applied to the data with the goal to solve a problem such as object classification. The level of DL performance is greater than classical ML and does not require a human to identify and compute the critical features. Instead, during training, DL algorithms “learn” discriminatory features that best predict the outcomes. The amount of human effort required to train DL systems is less because it requires no feature engineering, or computation. When it comes to the medical image analysis domain, the data sets are often inadequate to reach the full potential of DL. In the computer vision domain, transfer learning and fine tuning are often used to solve the problem of a small data set. In general, DL algorithms recognize the important features of images and properly give weight to these features by modulating their inner parameters to make predictions for new data, thus accomplishing identification, segmentation, classification, or grading, and demonstrating strong processing ability and intact information retention.
The superiority of CAD based on deep learning has recently been reported for a wide spectrum of diseases, including gastric cancer, diabetic retinopathy, cardiac arrhythmia, skin cancer, and colorectal polyp. A wide variety of image types were explored in these studies, including pathological slides, electrocardiograms, and radiological images. A well-trained algorithm for a specific disease can increase the accuracy of diagnosis and working efficiency of physicians or medical experts, liberating them from repetitive tasks, as well as enhancing diagnostic accuracy, especially in the presence of subtle pathological changes that cannot be detected by visual assessment. DL algorithms can be optimized through the tuning of hyperparameters such as learning rate, network architectures, and activation functions. CAD based on DL thus has the potential to improve the performance of ART.
Convolutional neural networks (CNNs) or ConvNets are DL network architectures that have recently been employed successfully for image segmentation, classification, object detection and recognition tasks, shattering performance benchmarks in many challenging applications. Medical image analysis applications have heavily relied on feature engineering approaches, where algorithm pipelines are used to explicitly delineate structures of interest using segmentation algorithms to measure predefined features of these structures that are believed to be predictive, and to use these features to train models that predict patient outcomes. In contrast, the feature learning paradigm of CNNs adaptively learns to transform images into highly predictive features for a specific learning objective. The images and patient labels are presented to a network composed of interconnected layers of convolutional filters that highlight important patterns in the images, and the filters and other parameters of the network are mathematically adapted to minimize prediction error. Feature learning avoids biased a priori definition of features and does not require the use of segmentation algorithms that are often confounded by artifacts.
A CNN is comprised of multiple layers with neurons that process portions of an input image. The outputs of these neurons are tiled to form an overlap, which provides a filtered representation of the original image. This process is repeated for each layer until the final output is reached, which is typically the probabilities of predicted classes. The training of a CNN requires many iterations to optimize network parameters. During each iteration, a batch of samples is chosen at random from the input training set and undergoes forward-propagation through the network layers. In order to achieve optimal results, parameters within the network are updated through back-propagation to minimize a cost function. Once trained, a network can be applied on new or unseen data to obtain predictions. The main advantages of CNNs are that features can be automatically learned from a training set without the need for expert knowledge or hard-coding. The extracted features are relatively robust to image transformations or variations. In the field of medical imaging, CNNs have been mainly utilized for detection, segmentation, and classification. These tasks make up part of the CAD process flow, and the effective feature extraction or phenotyping of patients from EMR is a key step for potential further applications of the technology, such as the successful performance of ART using DL techniques, which has not been contemplated to date among experts in the field.
Due to the sequential nature of EMR or EHR data, there have been recently multiple promising works studying clinical events as sequential data. Many of them were inspired by works in natural language modeling, since sentences can be easily modeled as sequences of signals. There is a growing interest in predicting treatment prescription and individual patient outcomes by extracting information from these data using advanced analytics approaches. In particular the recent success of DL in image and natural language processing has encouraged the application of these state-of-the-art techniques to modeling clinical data as well. CNNs, such as Recurrent Neural Networks (RNNs), which have proven to be powerful in language modeling and machine translation, are more frequently applied to medical event data for predictive purposes, since natural language and medical records share the same sequential nature. DL and more specifically RNN have not been contemplated for use in improving the performance of ART, leaving an opening for significant new improvements in the field of ART through application of these technologies, such as that embodied in the disclosure of the present application.
A fundamental component of performing ART requires the stimulation of the ovary to produce multiple eggs. In a natural cycle, a typical woman makes one egg per month alternating between the two ovaries. With ART, the administration of exogenous gonadotropins, principally follicle stimulating hormones (e.g., FSH), will encourage each ovary to make on average 10 to 15 eggs that grow in the fluid-filled ovarian follicle. As the follicles grow, they become progressively more dependent on gonadotropins for continued development and survival. FSH promotes granulosa cell proliferation and differentiation, allowing the follicle to increase in size. The follicles grow from their resting size of 3-7 mm to 20 mm in size over a 10-day medication treatment during which the dose is adjusted based upon ovarian response. During the 10 days of medications, US is performed 3 to 4 times to measure the follicular size and monitor the response. The size of the follicle predicts the likelihood there is an egg in the follicle, the quality of the egg, and the likelihood that the egg is mature.
A critical component of ART success is creating the best quality eggs possible for an individual patient. The quality and maturity of the eggs is directly predictive for the likelihood the eggs fertilize to become embryos and is predictive for the quality of the embryos. The quality of the resultant embryo is predictive for the implantation rate which is defined as (the pregnancy per one embryo transfer) and the overall pregnancy rate (if a multiple embryo transfer is performed). The numbers and quality of oocytes available are critical factors of the success rates for ART.
One of the challenges in patient care is that the eggs do not all start at the same size and grow at the same rate. Therefore, the follicles will vary in size at any one time during the stimulation period. The timing of a patient's egg retrieval (time-to-event) is therefore based on trying to determine when the majority of the follicles are mature in size. Sometimes that requires pushing the ovarian stimulation longer to effectively over stimulate some of the follicles with the goal of getting the majority of the follicles in the mature range. This follicle monitoring technique is performed with a combination of transvaginal US and blood measurements of estradiol and progesterone. The success of ART would benefit from the automated connection and coordination of ultrasound imaging, follicle monitoring, size determination, counting, determination of hormone levels and cycle days to important clinical time-to-events such as follicular maturity, egg maturity, number of embryos, blastocyst embryo development, and pregnancy rates. DL has the potential to improve ART where a sparsity of patient data exists for optimal timing of follicle extraction and implantation.
Survival analysis is about predicting the time duration until an event occurs. Traditional survival modeling assumes the time durations follow an unknown distribution. The Cox proportional hazard model is among the most popular of these models. The Cox model and its extensions are built on the proportional hazards hypothesis which assumes that the hazard ratio between two instances is constant in time and a risk prediction is based on a linear combination of covariates. However, there are too many complex interactions in real world clinical applications such as ART. A more comprehensive survival model is needed to better fit clinical data with nonlinear risk functions. In addition, a patient's EHR is longitudinal in nature because health conditions evolve over time. Therefore, temporal information is needed in order to apply CNN for analyzing patient EMR. DL of patient EMR or EHR has the potential to improve the determination of timing for follicle extraction and implantation.
The need exists for improving the performance of ART through an efficient management of IVF. Through applied effort, ingenuity, and innovation, Applicant has identified a number of deficiencies of the conventional approach, including but not limited to limitation of US modality, manual and inaccurate determination of AFC, identifying the quality and maturity of eggs, and the inability to synthesize patient health records, patient biomarker diagnostics, and treatment regimens for the intelligent determination of optimal follicle extraction timing to improve implantation and pregnancy rates. Applicant has developed a solution that is embodied by the present invention, which is described in detail below.